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■ We study the discovery potential of the flavor-changing neutral coupling (FCNC) 



(N 



X 



htc of the Higgs boson and the top quark through the rare tree-body decay h — > Wbc 
at Muon colliders for a light Higgs boson with mass 114 < < 145 GeV. This decay 



Q^' mode may compete with the SM background induced by the hWW coupling in some 

CD : 

models with a tree-level htc coupling and with models that predict this coupling at 
I the one-loop level in the range 10 -2 — 10 _1 . A future muon collider could test the 

£Sj ' scalar FCNC decay t — > he via Higgs decay h — > t*c — > bW + c down to values of the 

coupling g tc = 0.5 (that are equivalent to BR(t — > he) ~ 5 x 10 ). The LHC could 



probe values of gt c one order of magnitude smaller, unless other processes beyond the 



Q\ ' SM appear that through intense multi jet activity may clutter the t — > he signal. 

o" 
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I. INTRODUCTION 



Top quark decays induced by flavor-changing neutral couplings (FCNC) constitute a 
direct and sensitive probe of new-physics effects at energy scales of a few hundred GeV 
. While this type of process is highly suppressed in the Standart Model (SM) due to 



the GIM mechanism 



4j, most of its extensions predict branching ratios for these decay 
modes that may be observable at the LHC [5j|. It has been suggested also that some Higgs- 
mediated FCNC top quark processes could be observed at the LHC f], 7| and linear colliders 
8|. These processes are considered to be the most likely to shed light on the nature of the 
electroweak symmetry breaking mechanism if the FCNC top quark-Higgs boson couplings 



are as big as expected in most models Our general aim in this paper is to study 

the possibility of detecting the htc FCNC interaction at Muon colliders. There are studies 
focused on FCNC Higgs decays in this kind of colliders via its heavy lepton channels [q]. The 
CERN LHC is expected to discover a light Higgs boson with a mass 114 < rrih < 145 GeV 
[h]] and will determine some of its properties such as its fermionic and bosonic decay modes 
and couplings llj. A e + e~ linear collider with a center of mass e nerg y above 500 GeV will 
be able to significantly improve these preliminary measurements 12|. In particular, it has 
been suggested that the decay t — > he may be detected if the Higgs boson is somewhat lighter 
than the top quark in a linear collider with 500 fb _1 integrated luminosity and = 500 
GeV for models that predict branching ratios of order 10~ 4 for this FCNC decay mode jsj]. 
In the present paper we address the question concerning the possibility that the FCNC htc 
coupling could be measured with a higher accuracy in a muon collider through the three- 
body decay mode h — > t*c — > Wbc for a light Higgs boson. In the SM, this decay mode is 
determined by the W pair production channel h — > WW* — > Wbc but we expect that the 
htc coupling is high enough to compete with the SM contribution. 

It has been estimated that a linear collider with 500 fb _1 integrated luminosity and a 
center of mass energy of 500 GeV will begin to be sensitive to the htc coupling through the 
decay t he for extensions of the SM that predict branching ratios of this decay mode 
of order 10~ 5 — 5 x 10~ 4 . This correspond to a range of the htc coupling constant X tc of 
order 0.06 — 0.4 (g tc between 0.02 and 0.1) jsj]. Models with a tree-level htc coupling have 
branching ratios of the order of 10~ 3 — 10~ 2 , while models that induce this coupling at the 
one loop level predict branching ratios of order 10 -5 — 10~ 4 for this FCNC channel jll, Q]. In 
particular, in the context of the MSSM the scalar t — > he decay would be greater than other 
channels like t -> eg, and SUSY-QCD effects could yield BR(t -> he) ~ 5 x 1(T 4 Q. Similar 
values are possible in the context of the two-Higgs-doublet model^yj]. At the LHC, discovery 



limits for this decay mode were calculated in Ref. [6] and are of the order 5 x 10 , but 
the hadronic background may complicate the analysis. In the case of the muon collider, we 
expect that the production cross section on resonance for the chain fi + fi~ — > h — > t*c — > Wbc 
may be high enough for some extensions of the SM in such a way that it can compete with 
the SM background induced by the hWW* coupling. We assume that a light Higgs boson 
will be observed at the LHC and examine in detail the discovery potential for its FCNC htc 
coupling at a Muon collider. 
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At the LHC, unfortunately it is quite difficult to observe the three-body decay h — *> Wbc 
due to the fact that the expected QCD background coming from 4 jet events is orders of 
magnitude larger than the signal coming from the htc or hWW vertices. In this case it will 
be necessary to look for the purely leptonic events in order to have a discovery channel for 



a light Higgs boson at the LHC 15|] 



II. S-CHANNEL HIGGS PRODUCTION AT THE MUON COLLIDER 

A Muon collider designed with the purpose of studying the Higgs resonance with a very 
fined tuned center-of-mass (CM) energy equal to could become an effective Higgs fac- 



tory where precise measurements of the Higgs mass and width could be achieved [la]. It is 
expected that such a machine could yield annual integrated luminosities of order 20 fb~ l . 
However, the luminosity depends on how broad the muon energy spectrum is allowed to be. 
Here we assume a very low (albeit supposedly feasible) energy spread of a few MeVs and an 
integrated luminosity of order lfb^ 1 . 

Given the very narrow width of a light (below the WW threshold) Higgs boson the 



16(. The 



production cross section is extremely sensitive to the energy of the colliding muons 
Muon collider is expected to have a Gaussian shape of the energy spectrum of the colliding 
beams with an rms deviation 

R=^. (1) 



Where oe is the energy spread of the beams, and v^i is their CM energy. Notice that for 
\fs = 10 2 GeV and erg = 7 MeV the rms deviation is R = 10~ 4 . Near the resonance region 
the production cross section of the s-channel process — > h — > X is 

where Y h is the total width of the Higgs boson and s = + p^+) 2 is the CM energy 
squared of the colliding muons. Neglecting Bremmstrahlung, we can write the effective 
cross section as the convolution of the cross section above with the Gaussian distribution of 
the CM energy centered at yfs = ra^flf]]: 

a = —== — / <j(x)exp t, ax (6) 

V^a E J V 2<ri ; 
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We can re-write the above formula as: 

4 2 rtmax -t 2 /R 2 

9 = ^ - lijj)BF(h X) ^ £ m * (4, 

where a = Th/m^ and the limits of integration tmaxitmin) have been taken as ±10~ 2 . 
These limits cover all the significant contribution from the Gaussian distribution when we 
consider values of R = 10~ 4 and a ~ 10~ 4 for the rms deviation and the width-to-mass ratio 
of the light Higgs respectively. 

In Fig. [1] we show the Muon collider resonant Higgs production: the two most important 
channels /U + /i~ — > h — > 66 and /i + /U~ — > h — > W + W*~~ as well as the total, which is obtained 
by setting BF(h — > X) = 1 in Eq. (]4]). As the Higgs mass approaches the Since the 
BF(W~ -> 6c) is of order 0.7 x |V c6 | 2 = 0.7x0.04 2 ~ 10" 3 we expect the -»• h -> H/+6c 

channel to be of order a few fb. 

In Fig. [2] we show the resonant Higgs production cross section of a light Higgs decaying 
to the W + bc state. An additional reduction factor of order 0.6 could be used to account 
for Bremmstrahlung 16j . Fig. [2] shows a very small cross section of order 1/6 that would 
hardly yield one event with the integrated luminosity we consider. Comparing with the 
more general h —>■ Wjj decay channel the Wbc contains a CKM V c b factor of order 10 -3 
that explains why this SM decay of the light Higgs is so suppressed. 

In other words, the Wjj mode is 10 3 times higher and can yield a significant number of 
events. In fact, the h — > WW* mode turns out to be as good probe as h — > 66 even for a 
light Higgs (the latter is a dominant decay channel but it has a substantial background) [lfjj . 

The effective htc coupling we use is defined as: 

£ = —j= g tc hic + h.c. (5) 
2v 2 

The effective coupling g tc is very small in the SM gf c M ~ 10~ 6 but could be several orders of 
magnitude higher in other models like the THDM where an ansatz gf c M ~ ^ /m t m r /Mw ~ 0.2 



171 ] . It has recently 



is considered assuming the FC couplings scale with the quark masses 
been estimated that the LHC could measure g tc down to the order of 0.04(6]. 

In Fig. [3] we show the production cross section for the FC process —>■ h — > tc —>■ 

W + bc for a range of the Higgs mass. There, the coupling g tc is taken as g tc = 1. For this 
size the contribution from the SM fi + {i~ — ► h — * WW*c — > W + bc is about two orders of 
magnitude smaller. Fig. [3] does not include the SM process; if we take g tc of order 0.1 the two 
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collider operating at A 




FIG. 2: The effective a(h — > W + bc) cross section for a Muon collider operating at = with 
a energy spread i? = 1(T 4 (AE ~ 7MeV). 

contributions become similar and we would have to include interference effects. However, as 
cross sections of order a few fb are deemed to provide too little statistics we can realize the 
coupling g tc has to be a least of order 0.5 to yield enough production. In this respect, the 
LHC has a much better sensitivity to this coupling than the Muon collider. 




top pair events at the 

n 
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FIG. 3: The effective a(h — ► tc — ► W + bc) cross section for a Muon collider operating at = 
with an energy spread -R = 10 -4 (AS ~ 7MeV). 

III. A MUON COLLIDER VS THE LHC 

As mentioned before, for a Higgs mass smaller than the Top mass, the decay t —>■ he — ► bbc 
could be used to measure the coupling g tc . With the high statistics o ; 
LHC a very good sensitivity could be reached for this coupling; Ref. 
limit < 0.04 for a total integrated luminosity of 100 fb^ 1 . 

However, this limit could be greatly weakened if other (beyond the SM) processes came 
into play that would produce the same experimental signature as the one studied by Ref. . 
Such signature is based on a final state Ivbbbj where a bb pair comes from the decay of the 
Higgs, and the other b jet (as well as the lepton and neutrino) comes from the decay of the 
t quark. 

Let us assume that supersymmetric partners are discovered at the LHC. It is well known 
that multi-jet events with high missing transverse energy are among the typical signals of 
these new resonances. Such is the case for some mSUGRA scenarios like the test points LM1 
or LM3 used by the CMS Collaboration to evaluate their physics performance (see Chapter 



13 in [18j|). Some of these processes have a total cross section around 50 pb that is one order 
of magnitude smaller than the total cross section for tt production. The analysis of Ref. (J 
requires high P T for the jets, leptons and missing P T , and by doing this they cut out a lot 
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of the tt cross section (as well as other processes with lower transverse momenta). On the 
other hand, such high cuts are not expected to affect as much mSUGRA processes which 
have very high transverse momenta. A detailed quantitative study of this question is beyond 
the scope of this paper. However, our claim is that the presence of this kind of background 
would not invalidate their analysis, but it would potentially weaken their limits. 



IV. CONCLUSIONS 

The Higgs FCNC h — > t*c — > W + bc may compete with the SM background induced by 
the hWW coupling in some models with a tree-level htc coupling and with models that 
predict this coupling at the one- loop level in the range 10~ 2 — 10 _1 . For a light Higgs 
boson of mass around 120 GeV a future muon collider could test the scalar FCNC decay 
t — » he via Higgs decay h — > t*c — > bW + c down to values of the coupling g tc = 0.5 (that 
are equivalent to BR(i — > he) ~ 5 x 10~ 3 ). The LHC could probe values of gtc one order 
of magnitude smaller, unless other processes beyond the SM appear that through intense 
multi jet activity may clutter the t — > he signal. 
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